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Evaluation of Madison Park PLATO 
Training on August 2000 BPS City Algebra 

Test Achievement 



Christopher F. Baum 
October 1, 2001 



This report presents empirical findings from the analysis of 85 Madison Park 
High School students’ performance on the BPS City Algebra Test (BPSCAT) 
in June and August 2000, and how their participation in JFY/Boston’s PLATO 
computer-based instruction in the intervening months may have affected their 
attainment. We determine that the number of PLATO modules completed has a 
positive and statistically significant effect on the change in test scores, and that 
the significance of this effect is robust to a number of factors. All analysis is 
conducted with Stata version 7.0. 

Modeling improvement in BPSCAT performance 

The analysis of these data is reasonably considered via ordinary least squares 
regression, given that the retake score may be viewed as an outcome of the 
original score and the intervening treatment. The basic model considered is 
therefore: 

AUGi = 0 O + 0i PL ATOM ODi + 0 2 JUNi + e* 

where AUGi is the i th student’s August 2000 BPSCAT score, JUNi is the 
corresponding June 2000 score, and PLATOMODi measures the number of 
PLATO modules completed in the interim. Empirical results from estimation of 
this model over the full 85-student dataset are shown on line 14 of the attached 
log file (Model 1). The estimated coefficient on PLATOMOD is highly signifi- 
cant, with a p- value of 0.017 1 and a point estimate of 0.73: each PLATO module 

'The p- value is the probability of observing a t-statistic of this magnitude if the null hypoth- 
esis (that the associated population coefficient is zero) was true. 
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completed leads to a 0.73 point increase in the August score, ceteris paribus. 

Evaluating the robustness of the empirical findings 

Given the wide range of values recorded for the dependent variable, there 
may be some doubt that the assumption of homoskedasticity (a constant vari- 
ance of the error process e) is appropriate. If the errors are heteroskedastic, the 
estimated standard errors of the regression coefficients will be biased. This 
issue may be resolved by reestimating the regression specifying that robust 
(heteroskedasticity-consistent) standard errors are to be calculated. These re- 
sults are shown on line 16 (Model 2), in which we see that the robust standard 
error for PLATOMOD is even smaller than the OLS counterpart, and the in- 
terval estimate for that coefficient commensurately smaller. 

There may be concern that inclusion of those students who did not complete 
any PLATO modules (but nevertheless spent some time using the system) may 
bias the results. To evaluate this concern, the regression is reestimated for the 
77 students who completed one or more PLATO modules (see line 18, Model 3). 
The point estimate of the PLATOMOD coefficient is larger-0.92 versus 0.73- 
and significant at better than the 99% level, with a regression R 2 of 0.1722: that 
is, over 17% of the variation in August test score is attributable to the model 
that takes account of the June score and the PLATO experience. 

We are also concerned, when OLS regression is employed, in the presence of 
outliers. While a greater variance of the explanatory variables is usually helpful 
in estimating precise estimates, severe outliers may distort the relationship. 
A tabulation of PLATOMOD shows that one student completed 48 modules, 
while the next most ambitious student completed only 26. We reestimate the 
relationship in Model 4 (line 20) excluding both zero values and the extreme' 
value of 48, over the remaining 76 students. The results are qualitatively similar, 
with an even greater point estimate for PLATOMOD (1.14 points per module 
completed), significant at greater than the 99% level. 

The effect of time spent with PLATO on the test score 

One possible critique of these findings would suggest that the mere time 
spent with the PLATO system will have an effect. While time spent with the 
system is surely positively correlated with the number of modules completed, 
we would like to establish that it is mastery of the material-and not merely 
time spent at the keyboard-that has had an effect on attainment. We first fit a 
model (line 26, Model 5) in which the number of modules mastered is replaced 
with the amount of time spent, in decimal hours, with the system. That model 
shows that time spent is not systematically related to the August score, after 
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controlling for the June score. This result is unchanged if robust standard errors 
are computed (Model 6, line 28). 

Alternatively, we might consider that time is another measure of the “PLATO 
effect,” and include time spent alongside the number of modules completed. In 
Model 7 (line 33), we see that this model decisively echos the earlier findings: 
the effects of students’ PLATO experience is related to the number of modules 
completed (PLATOMOD), and not to the time spent with the system. This 
result is also achieved if robust standard errors are computed (Model 8, line 35). 

Summary findings 

At any conventional level of statistical significance, the 85 students’ August 
2000 BPSCAT test scores may be judged to have been meaningfully influenced 
by their participation in PLATO computer-based instruction, when that partici- 
pation is quantified as the number of PLATO modules mastered. This finding is 
robust to a number of forms of the model, and to the presence of heteroskedas- 
ticity in the error distribution. It is my reasoned judgment that these results 
illustrate, beyond a reasonable doubt, that the average student’s use of PLATO 
meaningfully improved his or her test score in the August 2000 retake. Although 
the interval estimates of the magnitude of this effect are broad, due to the lim- 
ited sample size and high variance of the August 2000 scores, they decisively 
exclude zero, and allow us to objectively conclude that the use of PLATO was 
highly beneficial for the representative student. A graphical illustration of this 
phenomenon is provided in the attached Figure, which presents a smoothed ver- 
sion of Model 4’s predicted values for the sample values of PLATOMOD. The 
positive slope of this line is indicative of the general improvement in August 2000 
scores accruing to those students who made greater efforts to master PLATO 
modules. 
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Background 

Dr. Christopher F Baum is an associate professor of economics at Boston 
College. He joined the BC faculty after earning the Ph.D. in economics from 
The University of Michigan-Ann Arbor in 1977. Baum has taught econometrics 
at the Ph.D. level for the past 20 years, as well as undergraduate econometrics 
and computational economics, and has authored over 30 refereed publications in 
applied economics and finance, including several related to program evaluation. 
He directs the University’s Graduate Statistical Assistant Program, established 
in 2000, and is an associate editor of Computational Economics and The Stata 
Journal 
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log: :Rumelihisari : Stetson: [documents] : JFY-Boston : BPSCAT . log 

log type: text 

opened on: 1 Oct 2001, 20:05:30 

. use "BPSCAT .dta" , clear 



. * generate time measurement 

. gen time=real (substr (platotime, 1 , index (platotime, " : 11 ) -1) ) +real (substr (platotime , ind 
> ex (platotime, +1 ,.)) /60 
(4 missing values generated) 



. * descriptives 
. summ math* platomod time 



Variable 


Obs 


Mean 


Std. Dev. 


Min 


Max 


mathjunOO 


85 


32.04706 


14.36236 


6 


6 B . 5 


mathaugOO 


85 


51.97647 


21.82887 


10 


90 


platomod 


85 


7.894118 


7.739853 


0 


48 


time 


81 


10.28416 


4 .495312 


0 


19.16667 



* test model of retake as function of number of modules completed and orig score 



* ( 1 ) 

regress mathaugOO platomod mathjunOO 



Source | 


SS 


df 


MS 




Number of obs 
F ( 2, 82) 

Prob > F 
R-squared 
Adj R-squared 
Root MSE 




85 
6 . 25 
0 . 0030 
0 . 1324 
0 . 1112 
20.579 


Model | 
Residual | 


5297 . 99104 
34727.9619 


2 

82 


2648 . 99552 
423 . 511731 






Total | 


40025.9529 


84 


476.49944 




= 


mathaugOO | 


Coef . 


Std. 


Err . t 


p>it| 


[95% Conf. 


Interval] 



platomod 


. 7303854 


.3000386 


2.43 


0.017 


.1335131 


1.327258 


mathjunOO 


.3007251 


.1616903 


1.86 


0.066 


- .0209284 


. 6223785 


cons 


36.57337 


5.583881 


6.55 


0.000 


25.46525 


47.68149 



. * ( 2 ) 

. regress mathaugOO platomod mathjunOO, robust 



Regression with robust standard errors Number of obs « 85 

F ( 2, 82) = 10.32 
Prob >F = 0.0001 
R-squared = 0.1324 
Root MSE = 20.579 



mathaugOO 


1 Coef . 


Robust 
Std. Err. 


t 


p>|t| 


[95% Conf. 


Interval] 


platomod 


. 7303854 


.2437015 


3.00 


0.004 


.2455855 


1.215185 


mathjunOO 


.3007251 


.1760156 


1.71 


0.091 


- .0494261 


.6508762 


_cons 


36.57337 


5 . 594281 


6.54 


0.000 


25.44456 


47.70218 



17 . * (3) 

18 . regress mathaugOO 



platomod mathjunOO if platomod>0 



Source | 


SS 


df 


MS 


Model 


6393 .42796 


2 


3196.71398 


Residual 


30741.0915 


74 


415.420156 


Total 1 


37134.5195 


76 


488.612098 



Number of 


obs = 


77 


F ( 2 , 


74) = 


7.70 


Prob > F 


= 


0 . 0009 


R- squared 


= 


0.1722 


Adj R- squared = 


0 . 1498 


Root MSE 


= 


20.382 



mathaugOO 


Coef. 


Std. Err. 


t 


p>|t| 


[95% Conf. 


Interval] 


platomod 

mathjunOO 

_cons 


. 9216613 
.2896614 
34.02281 


.3151606 

.1636177 

5.772744 


2.92 

1.77 

5.89 


0.005 

0.081 

0.000 


.2936901 
- .036354 
22.52037 


1.549632 

.6156769 

45.52526 



19 . * (4) 



. regress mathaugOO 


platomod 


mathjunOO if platomod>0 


& platomod<48 




Source | 




SS 


df 


MS 


Number of obs = 
F ( 2 , 73) = 

Prob > F 
R-squared = 

Adj R-squared = 
Root MSE 


76 
7.28 
0 . 0013 
0.1663 
0.1435 
20.378 


Model 1 
Residual | 


6047. 

30315 


81182 

.0698 


2 

73 


3023 .90591 
415.274928 


Total 1 


36362 


. 8816 


75 


484.838421 



mathaugOO 


Coef. 


Std. Err. 


t 


p>|t| 


[95% Conf. 


Interval] 


platomod 


1 . 14167 


.3827197 


2.98 


0.004 


.378911 


1.90443 


mathjunOO 


.3024267 


.1640739 


1.84 


0.069 


- .0245721 


. 6294256 


_cons 


32 . 02899 


6 . 098194 


5.25 


0.000 


19.87531 


44 . 18267 



21 . regplot, tl ("Actual and predicted for 0 < PlatoMod < 48") saving (predval , replace) 

22 . 

23 . * test model of retake as function of time spent and orig score 

24 . 

25 . * (5) 

26 . regress mathaugOO time mathjunOO 



Source | SS df MS Number of obs = 81 



Model I 
Residual | 


3304.43459 

35373.4419 


2 1652.2173 

78 453.505666 




c \ ' o ) 

Prob > F 
R-squared 
Adj R-squared 
Root MSE 


= 0.0307 

= 0.0854 

o 0.0620 

= 21.296 


Total | 38677.8765 


80 483.473457 




mathaugOO 


Coef. 


Std. Err. t 


p>|t| 


[95% Conf. 


Interval] 


time 

mathjunOO 

_cons 


.5230609 

.4071025 

32.8949 


.5297404 0.99 
.1632792 2.49 
7.877176 4.18 


0.327 

0.015 

0.000 


- .5315712 
.0820387 
17.21265 


1.577693 
. 7321664 
48.57716 
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27 . * (6) 

28 . regress mathaugOO time mathjunOO, robust 



Regression with robust standard errors Number of obs = 81 

F ( 2, 78) = 3.69 
Prob > F = 0 . 0294 
R-squared = 0 . 0854 
Root MSE = 21.296 



mathaugOO 


Coef . 


Robust 
Std . Err. 


t 


p>|t| 


[95% Conf. 


Interval] 


time 

mathjunOO 

_cons 


.5230609 
. 4071025 
32.8949 


.5878957 
. 1667289 
8.285842 


0.89 

2.44 

3.97 


0.376 

0.017 

0.000 


- .6473495 
.0751707 
16.39906 


1.693471 

.7390343 

49.39075 



29 . 

30 . * add time spent to original model 

31 . 

32 . * (7) 

33 . regress mathaugOO platomod time mathjunOO 



Source | SS df MS Number of obs = 81 



Model 

Residual 


6176.95105 

32500.9255 


3 2058.98368 

77 422.089942 




C \ J , i i j 

Prob > F 
R-squared 
Adj R-squared 
Root MSE 


= 0.0037 

= 0.1597 

= 0.1270 

= 20.545 


r 

Total | 38677.8765 


80 483.473457 




mathaugOO 


Coef . 


Std. Err. t 


p>|t| 


[95% Conf. 


Interval] 


platomod 

time 

mathjunOO 

_cons 


. 8672367 
- . 0341183 
. 2982756 
34 . 95827 


.3324367 2.61 
.5538977 -0.06 
.1629525 1.83 
7.640491 4.58 


0.011 

0.951 

0.071 

0.000 


.2052708 
-1.13707 
- . 0262043 
19.74411 


1.529203 
1 . 068833 
.6227555 
50 . 17243 



34 . * (8) 

35 . regress mathaugOO platomod time mathjunOO, robust 



Regression with robust standard errors Number of obs = 81 

F ( 3, 77) = 7.59 
Prob > F = 0.0002 
R-squared = 0.1597 
Root MSE = 20.545 



mathaugOO 


Coef . 


Robust 
Std. Err. 


t 


p>|t| 


[95% Conf. 


Interval] 


platomod 


. 8672367 


.2798133 


3.10 


0.003 


.3100573 


1.424416 


time 


- . 0341183 


.6048935 


-0.06 


0.955 


-1.238615 


1.170378 


mathjunOO 


.2982756 


.1799907 


1.66 


0.102 


- .0601316 


.6566828 


cons 


34 . 95827 


8.534039 


4.10 


0.000 


17.96483 


51.95171 
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37 . log close 
log : 
log type: 
closed on: 



tRumelihisari : Stetson: [documents] : JFY-Boston:BPSCAT. log 
text 

1 OCt 2001, 20:05:35 
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data and fit for MathAugOO 





1 0 



Actual and predicted for 0 < PlatoMod < 48 
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